MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

从点云中检测3D对象是一项实用但充满挑战的任务，最近引起了越来越多的关注。在本文中，我们提出了针对3D对象检测的标签引导辅助训练方法（LG3D），该方法是增强现有3D对象检测器的功能学习的辅助网络。具体而言，我们提出了两个新型模块：一个标签 - 通道诱导器，该模块诱导器将框架中的注释和点云映射到特定于任务的表示形式和一个标签 - 知识式插曲器，该标签知识映射器有助于获得原始特征以获得检测临界表示。提出的辅助网络被推理丢弃，因此在测试时间没有额外的计算成本。我们对室内和室外数据集进行了广泛的实验，以验证我们的方法的有效性。例如，我们拟议的LG3D分别在SUN RGB-D和SCANNETV2数据集上将投票人员分别提高了2.5％和3.1％的地图。

本文的目的是通过应用深度学习方法作为计算最佳资本分配策略的工具来研究系统性风险措施的新方法学框架。在这个新框架下，可以将系统性风险措施解释为通过在汇总单个风险之前将资本分配给单个机构来确保总体机构确保汇总系统的最低现金。除了在非常有限的情况下，此问题没有明确的解决方案。深度学习在财务模型和风险管理中越来越受到关注，我们建议我们基于深度学习的算法解决风险措施的原始问题和双重问题，从而学习公平的风险分配。特别是，我们的双重问题方法涉及受众所周知的生成对抗网络（GAN）方法的启发，以及对radon-Nikodym衍生产品的新设计的直接估计。我们通过对该主题进行大量数值研究结束了论文，并提供了与系统性风险措施相关的风险分配的解释。在指数偏好的特定情况下，与最佳显式溶液作为基准相比，数值实验表明了所提出的算法的出色性能。

视觉变换器将每个图像分成具有固定长度的令牌序列，并以与自然语言处理中的单词相同的方式处理令牌。更多令牌通​​常会导致更好的性能，但计算成本显着增加。通过谚语“一张图片胜过千言万语”，我们的目标是通过制造长图像短而加速VIT模型。为此，我们提出了一种新颖的方法在推论期间自适应地分配令牌长度。具体而言，我们首先培养一种含有可调整化 - vit（Revit）的Vit模型，可以处理任何具有不同令牌长度的给定输入。然后，我们从Revit检索“令牌长度标签”，并使用它培训轻量级令牌长度分配（TLA）。令牌长度标签是最小的令牌，以分割Revit可以使REVIT可以进行正确的预测，并且学习TLA以基于这些标签分配最佳令牌长度。 TLA使REVIT能够在推理期间使用最小足够数量的令牌处理图像。因此，通过减少VIT模型中的令牌数字来提高推广速度。我们的方法是一般的，与现代视觉变压器架构兼容，可以显着减少计算扩展。我们在两个任务中验证了我们对多个代表性VIT模型（DEIT，LV-VIT和TIMESFRER）的效果（图像分类和动作识别）。

这项工作调查了神经架构搜索中的批量标准化（NAS）。具体来说，Frankle等人。发现培训Batchnorm只能实现非竞争性能。此外，陈等人。声称培训Batchnorm只能加快10次单次NAS超网关的培训。批判性地，没有努力理解1）为什么训练Batchnorm只能找到具有减少的超空网训练时间的表演井架构，而且2）列车-BN的超网和标准列车超空网之间有什么区别。我们首先显示列车-BN网络融合到神经切线内核制度，从理论上获得与所有参数的所有参数相同的训练动态。我们的证据支持索赔仅在超培训时间上训练Batchnorm。然后，我们经验披露了培训-BN的超标网络在其他运营商的卷曲中提供了优势，导致架构之间的不公平竞争。这是因为只有卷积运算符被附加到Batchnorm。通过实验，我们表明这种不公平性使得搜索算法容易选择具有卷积的模型。为了解决这个问题，我们通过在每个操作员上放置批处理层来引入搜索空间的公平性。然而，我们观察到Chen等人的性能预测因子。在新的搜索空间上不可应用。为此，我们提出了一种新颖的综合性能指标，从三个视角评估网络：源自Batchnorm的理论属性的表达性，培训和不确定性。我们展示了我们对多NAS基准的方法（NAS-BENCH101，NAS-BENCH-201）和搜索空间（飞镖搜索空间和MOBILENET搜索空间）的有效性。

变压器在计算机视觉任务中表现出很大的潜力。常见的信念是他们的注意力令牌混合器模块对他们的能力做出了贡献。但是，最近的作品显示了变压器中的基于关注的模块可以被空间MLP所取代，由此产生的模型仍然表现得很好。基于该观察，我们假设变压器的一般架构，而不是特定的令牌混音器模块对模型的性能更为必要。为了验证这一点，我们刻意用尴尬的简单空间池汇集操作员取代变压器中的注意模块，以仅进行最基本的令牌混合。令人惊讶的是，我们观察到，派生模型称为池，在多台计算机视觉任务上实现了竞争性能。例如，在ImageNet-1K上，泳池制造器实现了82.1％的前1个精度，超越了调节的视觉变压器/ MLP样基线Deit-B / ResmmP-B24，比参数的35％/ 52％的准确度为0.3％/ 1.1％和48％/ 60％的Mac。泳道的有效性验证了我们的假设，并敦促我们启动“MetaFormer”的概念，这是一个从变压器抽象的一般架构，而无需指定令牌混音器。基于广泛的实验，我们认为MetaFormer是在视觉任务上实现最近变压器和MLP样模型的优越结果的关键球员。这项工作要求更具未来的研究，专门用于改善元形器，而不是专注于令牌混音器模块。此外，我们提出的池更换器可以作为未来的MetaFormer架构设计的起始基线。代码可在https://github.com/sail-sg/poolformer使用

Recently, contrastive learning (CL) has emerged as a successful method for unsupervised graph representation learning. Most graph CL methods first perform stochastic augmentation on the input graph to obtain two graph views and maximize the agreement of representations in the two views. Despite the prosperous development of graph CL methods, the design of graph augmentation schemes-a crucial component in CL-remains rarely explored. We argue that the data augmentation schemes should preserve intrinsic structures and attributes of graphs, which will force the model to learn representations that are insensitive to perturbation on unimportant nodes and edges. However, most existing methods adopt uniform data augmentation schemes, like uniformly dropping edges and uniformly shuffling features, leading to suboptimal performance. In this paper, we propose a novel graph contrastive representation learning method with adaptive augmentation that incorporates various priors for topological and semantic aspects of the graph. Specifically, on the topology level, we design augmentation schemes based on node centrality measures to highlight important connective structures. On the node attribute level, we corrupt node features by adding more noise to unimportant node features, to enforce the model to recognize underlying semantic information. We perform extensive experiments of node classification on a variety of real-world datasets. Experimental results demonstrate that our proposed method consistently outperforms existing state-of-the-art baselines and even surpasses some supervised counterparts, which validates the effectiveness of the proposed contrastive framework with adaptive augmentation. CCS CONCEPTS• Computing methodologies → Unsupervised learning; Neural networks; Learning latent representations.

Increasing research interests focus on sequential recommender systems, aiming to model dynamic sequence representation precisely. However, the most commonly used loss function in state-of-the-art sequential recommendation models has essential limitations. To name a few, Bayesian Personalized Ranking (BPR) loss suffers the vanishing gradient problem from numerous negative sampling and predictionbiases; Binary Cross-Entropy (BCE) loss subjects to negative sampling numbers, thereby it is likely to ignore valuable negative examples and reduce the training efficiency; Cross-Entropy (CE) loss only focuses on the last timestamp of the training sequence, which causes low utilization of sequence information and results in inferior user sequence representation. To avoid these limitations, in this paper, we propose to calculate Cumulative Cross-Entropy (CCE) loss over the sequence. CCE is simple and direct, which enjoys the virtues of painless deployment, no negative sampling, and effective and efficient training. We conduct extensive experiments on five benchmark datasets to demonstrate the effectiveness and efficiency of CCE. The results show that employing CCE loss on three state-of-the-art models GRU4Rec, SASRec, and S3-Rec can reach 125.63%, 69.90%, and 33.24% average improvement of full ranking NDCG@5, respectively. Using CCE, the performance curve of the models on the test data increases rapidly with the wall clock time, and is superior to that of other loss functions in almost the whole process of model training.

In the scenario of black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful adversarial perturbation based on query feedback under a query budget. Due to the limited feedback information, existing query-based black-box attack methods often require many queries for attacking each benign example. To reduce query cost, we propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. Specifically, by treating the attack on each benign example as one task, we develop a meta-learning framework by training a meta-generator to produce perturbations conditioned on benign examples. When attacking a new benign example, the meta generator can be quickly fine-tuned based on the feedback information of the new task as well as a few historical attacks to produce effective perturbations. Moreover, since the meta-train procedure consumes many queries to learn a generalizable generator, we utilize model-level adversarial transferability to train the meta-generator on a white-box surrogate model, then transfer it to help the attack against the target model. The proposed framework with the two types of adversarial transferability can be naturally combined with any off-the-shelf query-based attack methods to boost their performance, which is verified by extensive experiments.

Supervised Deep-Learning (DL)-based reconstruction algorithms have shown state-of-the-art results for highly-undersampled dynamic Magnetic Resonance Imaging (MRI) reconstruction. However, the requirement of excessive high-quality ground-truth data hinders their applications due to the generalization problem. Recently, Implicit Neural Representation (INR) has appeared as a powerful DL-based tool for solving the inverse problem by characterizing the attributes of a signal as a continuous function of corresponding coordinates in an unsupervised manner. In this work, we proposed an INR-based method to improve dynamic MRI reconstruction from highly undersampled k-space data, which only takes spatiotemporal coordinates as inputs. Specifically, the proposed INR represents the dynamic MRI images as an implicit function and encodes them into neural networks. The weights of the network are learned from sparsely-acquired (k, t)-space data itself only, without external training datasets or prior images. Benefiting from the strong implicit continuity regularization of INR together with explicit regularization for low-rankness and sparsity, our proposed method outperforms the compared scan-specific methods at various acceleration factors. E.g., experiments on retrospective cardiac cine datasets show an improvement of 5.5 ~ 7.1 dB in PSNR for extremely high accelerations (up to 41.6-fold). The high-quality and inner continuity of the images provided by INR has great potential to further improve the spatiotemporal resolution of dynamic MRI, without the need of any training data.